Near-Optimal Herding

The Herding algorithm is an algorithm of recent interest in the machine learning community, motivated by inference in Markov random fields. It solves the following sampling problem: given a set X ⊂ R with mean μ, construct an infinite sequence of points from X such that, for every t ≥ 1, the mean of the first t points in that sequence lies within Euclidean distance O(1/t) of μ. The classic Perceptron boundedness theorem implies that such a result actually holds for a wide class of algorithms, although the factors suppressed by the O(1/t) notation are exponential in d. Thus, to establish a non-trivial result for the sampling problem, one must carefully analyze the factors suppressed by the O(1/t) error bound. This paper studies the best error that can be achieved for the sampling problem. Known analyses of the Herding algorithm give a error bound that depends on geometric properties ofX but, even under favorable conditions, this bound depends linearly on d. We present a new polynomialtime algorithm that solves the sampling problem with error O (√ d log|X |/t ) . Our algorithm is based on recent algorithmic results in discrepancy theory. We also show that any algorithm for the sampling problem must have error Ω( √ d/t), implying that our algorithm is optimal to within logarithmic factors.